| Response category | Number of responses | Percent |
|---|---|---|
| 0 | 17 | 1.1 |
| 1 | 87 | 5.4 |
| 2 | 150 | 9.3 |
| 3 | 354 | 22.0 |
| 4 | 527 | 32.7 |
| 5 | 387 | 24.0 |
| 6 | 88 | 5.5 |
Total number of responses for all items
1 Why Rasch analysis in R?
- Traceability
- Transparency in decisionmaking
- Reproducibility
- Freely available
- Documentation in the same file as analysis
- (with Quarto/Rmarkdown)
1.1 Agenda overview
- Sample analysis with provided data and code
- WAAQ-7
- PSS-10
- Suggested structure for analysis
- Confirmatory and/or exploratory
1.2 Quarto formatting basics
https://quarto.org/docs/authoring/markdown-basics.html
- Keep the page above open in a web browser
- We will use library(kableExtra) for tables
- since it works with HTML/revealjs and PDF output
- even though library(formattable) looks nicer…
1.3 Exercise 1
- I’ll walk you through the Rstudio interface
- You’ll run the setup code chunk, and practice navigation a bit
- Enable/disable Outline
- Navigate with Outline
- Navigate with chunks
- Switch between source and visual editor
- Changing size of the quadrant windows
- Look at the objects df and itemlabels
1.4 Exercise 2
- We will walk through the code chunks together
- Learn what they do, but not how (in detail)
- I’ll explain some basic R concepts as we move ahead
- Focus on how you can structure the analysis
- Discuss the output briefly
- feedback on how to improve is welcome :)
- Small exercises are described at the end of the file
1.5 Exercise 3 (later on)
- In the folder “Exercise2” you will find a RaschR2 qmd file
- It contains a setup block that will import a dataset (PSS-10)
- You can practice using this file (RaschR1) as a template
- copy code chunks for your analysis from this file to the other file
- copy code, run the chunks, check the output, write commentary. Repeat
- there is a folder called “Exercise2solution” that shows how I did the analysis
2 Background information
The Work-related Acceptance and Action Questionnaire (Bond, Lloyd, and Guenole 2013) is a measure of psychological flexibility contextualized to a workplace setting.
The WAAQ uses 7 questions, all using the same response scale with 7 steps (ranging from “Stämmer aldrig” to “Stämmer alltid”, with labels on all categories).
| itemnr | item |
|---|---|
| WAAQ_1 | Jag kan arbeta effektiv trots personliga problem som oroar mig |
| WAAQ_2 | Jag kan erkänna mina misstag i arbetet och ändå vara framgångsrik |
| WAAQ_3 | Jag kan arbeta mycket effektivt även när jag är orolig för någonting. |
| WAAQ_4 | Oro hindrar mig inte från att nå framgångar. |
| WAAQ_5 | Jag kan göra vad som krävs, oavsett hur jag mår. |
| WAAQ_6 | Jag kan arbeta effektivt även när jag tvivlar på mig själv. |
| WAAQ_7 | Mina tankar och känslor hindrar mig inte i mitt arbete. |
Items in the Swedish WAAQ
2.1 Participants
We have 230 participants and their gender distribution is shown in the table below. All participants are Swedish adults and the dataset was collected to validate a measure of social interactions at workplaces (Johansson 2021; Johansson and Biglan 2021).
| Gender | n | Percent |
|---|---|---|
| Female | 76 | 33.2 |
| Male | 153 | 66.8 |
2.2 Descriptives of raw data
Responses to all items are summarized below.
2.3 Descriptives - item level
2.3.1 Recoding responses
There are fewer than 10 responses in category 0 for all items, which is problematic for the analysis. We will merge the two lowest categories. For item 2 the four lowest categories will be merged.
Some cells still have fewer than 10 responses. Let’s look at the response categories using Rasch modeling to produce ICC plots.
3 Analysis of response categories
Items 1 and 5 have issues with disordered response categories, which need to be addressed. For both items the two lowest categories will be merged.
3.1 ICC plots for recoded items
Response categories are working as expected, after the adjustments made.
3.2 Floor/ceiling effects - raw data
4 Rasch analysis 1
The eRm package, which uses Conditional Maximum Likelihood (CML) estimation, will be used primarily. For this analysis, the Partial Credit Model will be used.
| OutfitMSQ | InfitMSQ | OutfitZSTD | InfitZSTD | |
|---|---|---|---|---|
| WAAQ_1 | 0.77 | 0.793 | -2.614 | -2.379 |
| WAAQ_2 | 1.371 | 1.3 | 3.717 | 3.158 |
| WAAQ_3 | 0.667 | 0.653 | -3.881 | -4.077 |
| WAAQ_4 | 0.963 | 0.912 | -0.366 | -0.937 |
| WAAQ_5 | 0.866 | 0.829 | -1.383 | -1.995 |
| WAAQ_6 | 0.849 | 0.774 | -1.66 | -2.637 |
| WAAQ_7 | 0.817 | 0.803 | -2.003 | -2.217 |
| Eigenvalues |
|---|
| 1.51 |
| 1.38 |
| 1.18 |
| 1.10 |
| 1.01 |
5 1st contrast loadings
| WAAQ_1 | WAAQ_2 | WAAQ_3 | WAAQ_4 | WAAQ_5 | WAAQ_6 | WAAQ_7 | |
|---|---|---|---|---|---|---|---|
| WAAQ_1 | |||||||
| WAAQ_2 | -0.06 | ||||||
| WAAQ_3 | 0.09 | -0.24 | |||||
| WAAQ_4 | -0.15 | -0.06 | -0.16 | ||||
| WAAQ_5 | -0.13 | -0.25 | -0.16 | -0.16 | |||
| WAAQ_6 | -0.21 | -0.18 | -0.13 | -0.25 | -0.12 | ||
| WAAQ_7 | -0.27 | -0.28 | -0.09 | -0.24 | -0.06 | -0.07 | |
| Note: | |||||||
| Relative cut-off value (highlighted in red) is 0.049, which is 0.2 above the average correlation. |
| itemnr | item |
|---|---|
| WAAQ_1 | Jag kan arbeta effektiv trots personliga problem som oroar mig |
| WAAQ_2 | Jag kan erkänna mina misstag i arbetet och ändå vara framgångsrik |
| WAAQ_3 | Jag kan arbeta mycket effektivt även när jag är orolig för någonting. |
| WAAQ_4 | Oro hindrar mig inte från att nå framgångar. |
| WAAQ_5 | Jag kan göra vad som krävs, oavsett hur jag mår. |
| WAAQ_6 | Jag kan arbeta effektivt även när jag tvivlar på mig själv. |
| WAAQ_7 | Mina tankar och känslor hindrar mig inte i mitt arbete. |
Values in red are below/above the pre-set cutoff values.
A correlation matrix is created based on the Rasch model residuals for each item. The average correlation is calculated, and item pairs that correlate more than the pre-set cutoff value (see footnote) above the average correlation are indicated in red in the table below.
Items 1 and 3 are correlated above the relative cutoff value. Looking at the item descriptions, both are related to working efficiently when being worried in general (item 3) or being worried about personal problems. As we saw in the section on item fit, item 3 had quite low item fit values. This is a common issue with broad questions.
It would be reasonable to eliminate either question 1 or 3, since they clearly overlap in content. We will leave this for an exercise.
6 DIF analysis
Our sample is too small to conduct a properly powered DIF-analysis of any of the demographic variables available (gender/age). N > 250 per subgroup would be ideal, and N > 150 minimal.
For the sake of practice, we will go through the steps in setting up and conducting a DIF analysis anyway.
We will use the Rasch Tree tool (Strobl, Kopf, and Zeileis 2013), which makes it possible to evaluate interaction DIF, if sample size allows. For this analysis, we will only investigate gender effects on DIF.
[1] "No significant DIF found."No DIF was found in our small sample.
7 Targeting & item/person locations
7.1 Reliability
“Test information” shows the collective information based on the combination of all items into a unidimensional scale.
The horizontal lines at TIF 3.33 and 5.0 indicate the conventional cutoff values for acceptable reliability of 0.7 and 0.8, respectively.
The table footnote includes information on floor/ceiling effects.
7.1.1 Item information
“Item information” shows individual curves indicating the amount of information contributed by each item.
7.2 Person location and infit ZSTD
8 Item parameters
| Threshold 1 | Threshold 2 | Threshold 3 | Threshold 4 | Threshold 5 | Item location | |
|---|---|---|---|---|---|---|
| WAAQ_1 | -1.60 | -0.58 | 1.40 | 4.87 | NA | 1.02 |
| WAAQ_2 | -1.22 | 0.69 | 3.86 | NA | NA | 1.11 |
| WAAQ_3 | -2.04 | -1.67 | 0.19 | 2.20 | 5.81 | 0.9 |
| WAAQ_4 | -1.87 | -0.96 | -0.10 | 2.25 | 4.32 | 0.73 |
| WAAQ_5 | -0.44 | 0.49 | 2.33 | 4.40 | NA | 1.69 |
| WAAQ_6 | -1.43 | -0.52 | 0.33 | 2.60 | 5.04 | 1.2 |
| WAAQ_7 | -1.26 | -0.75 | 0.08 | 1.74 | 4.44 | 0.85 |
9 Ordinal to interval transformation table
| Ordinal sum score | Logits | SE (Logits) | Interval score (0-100) | SE (0-100) | MU (0-100) |
|---|---|---|---|---|---|
| 1 | -3.45 | 1.01 | 0.0 | 9.7 | 19.4 |
| 2 | -2.72 | 0.73 | 7.0 | 7.0 | 14.0 |
| 3 | -2.28 | 0.62 | 11.2 | 6.0 | 12.0 |
| 4 | -1.94 | 0.55 | 14.4 | 5.3 | 10.6 |
| 5 | -1.66 | 0.51 | 17.1 | 4.9 | 9.8 |
| 6 | -1.41 | 0.49 | 19.5 | 4.7 | 9.4 |
| 7 | -1.18 | 0.47 | 21.7 | 4.5 | 9.0 |
| 8 | -0.97 | 0.46 | 23.7 | 4.4 | 8.8 |
| 9 | -0.76 | 0.45 | 25.7 | 4.3 | 8.6 |
| 10 | -0.56 | 0.45 | 27.7 | 4.3 | 8.6 |
| 11 | -0.36 | 0.45 | 29.6 | 4.3 | 8.6 |
| 12 | -0.15 | 0.45 | 31.6 | 4.3 | 8.6 |
| 13 | 0.06 | 0.46 | 33.6 | 4.4 | 8.8 |
| 14 | 0.28 | 0.47 | 35.7 | 4.5 | 9.0 |
| 15 | 0.50 | 0.49 | 37.8 | 4.7 | 9.4 |
| 16 | 0.75 | 0.50 | 40.2 | 4.8 | 9.6 |
| 17 | 1.01 | 0.52 | 42.7 | 5.0 | 10.0 |
| 18 | 1.28 | 0.53 | 45.3 | 5.1 | 10.2 |
| 19 | 1.58 | 0.55 | 48.1 | 5.3 | 10.6 |
| 20 | 1.89 | 0.57 | 51.1 | 5.5 | 11.0 |
| 21 | 2.23 | 0.58 | 54.4 | 5.6 | 11.2 |
| 22 | 2.58 | 0.60 | 57.7 | 5.8 | 11.6 |
| 23 | 2.95 | 0.61 | 61.2 | 5.9 | 11.8 |
| 24 | 3.33 | 0.63 | 64.9 | 6.1 | 12.2 |
| 25 | 3.73 | 0.64 | 68.7 | 6.2 | 12.4 |
| 26 | 4.16 | 0.66 | 72.8 | 6.4 | 12.8 |
| 27 | 4.61 | 0.69 | 77.1 | 6.7 | 13.4 |
| 28 | 5.11 | 0.74 | 81.9 | 7.1 | 14.2 |
| 29 | 5.72 | 0.84 | 87.8 | 8.1 | 16.2 |
| 31 | 7.00 | NA | 100.0 | NA | NA |
| Note: | |||||
| SE = Standard Error. Multiply by 2 for MU = Measurement Uncertainty |
10 Item location & infit ZSTD
11 Item location & outfit ZSTD
12 First contrast loadings & item locations
The figure above shows the item loadings on the first contrast factor of the PCA of Rasch model residuals, and the item locations.
13 Software used
| Package | Version | Citation |
|---|---|---|
| base | 4.2.1 | R Core Team (2022) |
| car | 3.1.0 | Fox and Weisberg (2019) |
| cowplot | 1.1.1 | Wilke (2020) |
| eRm | 1.0.2 | Mair and Hatzinger (2007b); Mair and Hatzinger (2007a); Hatzinger and Rusch (2009); Rusch, Maier, and Hatzinger (2013); Koller, Maier, and Hatzinger (2015); Debelak and Koller (2019); Mair, Hatzinger, and Maier (2021) |
| formattable | 0.2.1 | Ren and Russell (2021) |
| grateful | 0.1.11 | Rodríguez-Sánchez, Jackson, and Hutchins (2022) |
| HH | 3.1.49 | Heiberger and Holland (2004); Heiberger and Robbins (2014); Heiberger and Holland (2015); Heiberger (2022) |
| kableExtra | 1.3.4 | Zhu (2021) |
| knitr | 1.40 | Xie (2014); Xie (2015); Xie (2022) |
| matrixStats | 0.62.0 | Bengtsson (2022) |
| mirt | 1.37.1 | Chalmers (2012) |
| mokken | 3.0.6 | Van der Ark (2007); Van der Ark (2012) |
| psych | 2.2.5 | Revelle (2022) |
| psychotree | 0.16.0 | Trepte and Verbeet (2010); Strobl, Wickelmaier, and Zeileis (2011); Strobl, Kopf, and Zeileis (2015); Komboz, Zeileis, and Strobl (2018); Wickelmaier and Zeileis (2018) |
| reshape | 0.8.9 | Wickham (2007) |
| RISEkbmRasch | 0.1.4 | Johansson (2022) |
| rmarkdown | 2.16 | Xie, Allaire, and Grolemund (2018); Xie, Dervieux, and Riederer (2020); Allaire et al. (2022) |
| scales | 1.2.1 | Wickham and Seidel (2022) |
| tidyverse | 1.3.2 | Wickham et al. (2019) |
14 Exercises
- Make a table of partipant age distribution
- Eliminate either item 1 or 3 and check the effects on reliability
- hint: df.omit.na$item <- NULL
- Bonus if you are quick:
- Run a DIF analysis using age as DIF variable
- first add the dif.age variable in the setup chunk
- then duplicate the DIF chunk and modify code
15 References
Allaire, JJ, Yihui Xie, Jonathan McPherson, Javier Luraschi, Kevin Ushey, Aron Atkins, Hadley Wickham, Joe Cheng, Winston Chang, and Richard Iannone. 2022. Rmarkdown: Dynamic Documents for r. https://github.com/rstudio/rmarkdown.
Bengtsson, Henrik. 2022. matrixStats: Functions That Apply to Rows and Columns of Matrices (and to Vectors). https://CRAN.R-project.org/package=matrixStats.
Bond, Frank W., Joda Lloyd, and Nigel Guenole. 2013. “Work-Related Acceptance and Action Questionnaire.” American Psychological Association (APA). https://doi.org/10.1037/t72150-000.
Chalmers, R. Philip. 2012. “mirt: A Multidimensional Item Response Theory Package for the R Environment.” Journal of Statistical Software 48 (6): 1–29. https://doi.org/10.18637/jss.v048.i06.
Debelak, Rudolf, and Ingrid Koller. 2019. “Testing the Local Independence Assumption of the Rasch Model With Q3-Based Nonparametric Model Tests.” Applied Psychological Measurement. https://doi.org/10.1177/0146621619835501.
Fox, John, and Sanford Weisberg. 2019. An R Companion to Applied Regression. Third. Thousand Oaks CA: Sage. https://socialsciences.mcmaster.ca/jfox/Books/Companion/.
Hatzinger, Reinhold, and Thomas Rusch. 2009. “IRT models with relaxed assumptions in eRm: A manual-like instruction.” Psychology Science Quarterly 51.
Heiberger, Richard M. 2022. HH: Statistical Analysis and Data Display: Heiberger and Holland. https://CRAN.R-project.org/package=HH.
Heiberger, Richard M., and Burt Holland. 2004. Statistical Analysis and Data Display: An Intermediate Course with Examples in S-Plus, R, and SAS. First. Springer-Verlag, New York. https://link.springer.com/book/10.1007/978-1-4757-4284-8.
———. 2015. Statistical Analysis and Data Display: An Intermediate Course with Examples in R. Second. Springer-Verlag, New York. https://link.springer.com/book/10.1007/978-1-4939-2122-5.
Heiberger, Richard M., and Naomi B. Robbins. 2014. “Design of Diverging Stacked Bar Charts for Likert Scales and Other Applications.” Journal of Statistical Software 57 (5): 1–32. https://doi.org/10.18637/jss.v057.i05.
Johansson, Magnus. 2021. “Dataset for the Initial Development of the Group Nurturance Inventory.” figshare. https://doi.org/10.6084/M9.FIGSHARE.13042010.V3.
———. 2022. RISEkbmRasch: Psychometric Analysis with Rasch Measurement Theory. https://github.com/pgmj/RISEkbmRasch.
Johansson, Magnus, and Anthony Biglan. 2021. “The Group Nurturance Inventory Initial Psychometric Evaluation Using Rasch and Factor Analysis.” BMC Public Health 21 (1). https://doi.org/10.1186/s12889-021-11474-5.
Koller, Ingrid, Marco Johannes Maier, and Reinhold Hatzinger. 2015. “An Empirical Power Analysis of Quasi-Exact Tests for the Rasch Model: Measurement Invariance in Small Samples.” Methodology 11. https://doi.org/10.1027/1614-2241/a000090.
Komboz, Basil, Achim Zeileis, and Carolin Strobl. 2018. “Tree-Based Global Model Tests for Polytomous Rasch Models.” Educational and Psychological Measurement 78 (1): 128–66. https://doi.org/10.1177/0013164416664394.
Mair, Patrick, and Reinhold Hatzinger. 2007a. “CML based estimation of extended Rasch models with the eRm package in R.” Psychology Science 49.
———. 2007b. “Extended Rasch modeling: The eRm package for the application of IRT models in R.” Journal of Statistical Software 20. https://www.jstatsoft.org/v20/i09.
Mair, Patrick, Reinhold Hatzinger, and Marco Johannes Maier. 2021. eRm: Extended Rasch Modeling. https://cran.r-project.org/package=eRm.
R Core Team. 2022. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation for Statistical Computing. https://www.R-project.org/.
Ren, Kun, and Kenton Russell. 2021. Formattable: Create ’Formattable’ Data Structures. https://CRAN.R-project.org/package=formattable.
Revelle, William. 2022. Psych: Procedures for Psychological, Psychometric, and Personality Research. Evanston, Illinois: Northwestern University. https://CRAN.R-project.org/package=psych.
Rodríguez-Sánchez, Francisco, Connor P. Jackson, and Shaurita D. Hutchins. 2022. Grateful: Facilitate Citation of r Packages. https://github.com/Pakillo/grateful.
Rusch, Thomas, Marco Johannes Maier, and Reinhold Hatzinger. 2013. “Linear logistic models with relaxed assumptions in R.” In Algorithms from and for Nature and Life, edited by Berthold Lausen, Dirk van den Poel, and Alfred Ultsch. Studies in Classification, Data Analysis, and Knowledge Organization. New York: Springer. https://doi.org/10.1007/978-3-319-00035-0_34.
Strobl, Carolin, Julia Kopf, and Achim Zeileis. 2013. “Rasch Trees: A New Method for Detecting Differential Item Functioning in the Rasch Model.” Psychometrika 80 (2): 289–316. https://doi.org/10.1007/s11336-013-9388-3.
———. 2015. “Rasch Trees: A New Method for Detecting Differential Item Functioning in the Rasch Model.” Psychometrika 80 (2): 289–316. https://doi.org/10.1007/s11336-013-9388-3.
Strobl, Carolin, Florian Wickelmaier, and Achim Zeileis. 2011. “Accounting for Individual Differences in Bradley-Terry Models by Means of Recursive Partitioning.” Journal of Educational and Behavioral Statistics 36 (2): 135–53. https://doi.org/10.3102/1076998609359791.
Trepte, Sabine, and Markus Verbeet, eds. 2010. Allgemeinbildung in Deutschland – Erkenntnisse Aus Dem SPIEGEL Studentenpisa-Test. Wiesbaden: VS Verlag.
Van der Ark, L. Andries. 2007. “Mokken Scale Analysis in R.” Journal of Statistical Software 20 (11): 1–19. https://www.jstatsoft.org/article/view/v020i11.
———. 2012. “New Developments in Mokken Scale Analysis in R.” Journal of Statistical Software 48 (5): 1–27. https://www.jstatsoft.org/article/view/v048i05.
Wickelmaier, Florian, and Achim Zeileis. 2018. “Using Recursive Partitioning to Account for Parameter Heterogeneity in Multinomial Processing Tree Models.” Behavior Research Methods 50 (3): 1217–33. https://doi.org/10.3758/s13428-017-0937-z.
Wickham, Hadley. 2007. “Reshaping Data with the Reshape Package.” Journal of Statistical Software 21 (12). https://www.jstatsoft.org/v21/i12/.
Wickham, Hadley, Mara Averick, Jennifer Bryan, Winston Chang, Lucy D’Agostino McGowan, Romain François, Garrett Grolemund, et al. 2019. “Welcome to the tidyverse.” Journal of Open Source Software 4 (43): 1686. https://doi.org/10.21105/joss.01686.
Wickham, Hadley, and Dana Seidel. 2022. Scales: Scale Functions for Visualization. https://CRAN.R-project.org/package=scales.
Wilke, Claus O. 2020. Cowplot: Streamlined Plot Theme and Plot Annotations for ’Ggplot2’. https://CRAN.R-project.org/package=cowplot.
Xie, Yihui. 2014. “Knitr: A Comprehensive Tool for Reproducible Research in R.” In Implementing Reproducible Computational Research, edited by Victoria Stodden, Friedrich Leisch, and Roger D. Peng. Chapman; Hall/CRC. http://www.crcpress.com/product/isbn/9781466561595.
———. 2015. Dynamic Documents with R and Knitr. 2nd ed. Boca Raton, Florida: Chapman; Hall/CRC. https://yihui.org/knitr/.
———. 2022. Knitr: A General-Purpose Package for Dynamic Report Generation in r. https://yihui.org/knitr/.
Xie, Yihui, J. J. Allaire, and Garrett Grolemund. 2018. R Markdown: The Definitive Guide. Boca Raton, Florida: Chapman; Hall/CRC. https://bookdown.org/yihui/rmarkdown.
Xie, Yihui, Christophe Dervieux, and Emily Riederer. 2020. R Markdown Cookbook. Boca Raton, Florida: Chapman; Hall/CRC. https://bookdown.org/yihui/rmarkdown-cookbook.
Zhu, Hao. 2021. kableExtra: Construct Complex Table with ’Kable’ and Pipe Syntax. https://CRAN.R-project.org/package=kableExtra.